[CANN] Optimize CANN buffer pool memory management#12875
Merged
hipudding merged 4 commits intoggml-org:masterfrom Apr 15, 2025
Merged
[CANN] Optimize CANN buffer pool memory management#12875hipudding merged 4 commits intoggml-org:masterfrom
hipudding merged 4 commits intoggml-org:masterfrom
Conversation
Contributor
Author
|
@thxCode Thank you very much for your contribution. Given the superiority of the recent code changes, we have decided to merge it into the main repository. We now invite you to review this PR and look forward to your response. |
Contributor
|
@bachelor-dou thanks, I think all things look good, and I would appreciate it if you could optimize them. |
hipudding
approved these changes
Apr 15, 2025
colout
pushed a commit
to colout/llama.cpp
that referenced
this pull request
Apr 21, 2025
Multiple optional memory pools are provided for CANN, including VMM, priority queue-based, and traditional memory pools. 1.When the memory pool is available and GGML_CANN_DISABLE_VMM_POOL is not defined, the VMM pool is selected by default. 2.Otherwise, if GGML_CANN_ENABLE_BUF_PRIO_POOL is defined, the priority queue-based memory pool is used. 3.If neither condition is met, the default memory pool is used.
timwu
pushed a commit
to timwu/llama.cpp
that referenced
this pull request
Dec 20, 2025
Multiple optional memory pools are provided for CANN, including VMM, priority queue-based, and traditional memory pools. 1.When the memory pool is available and GGML_CANN_DISABLE_VMM_POOL is not defined, the VMM pool is selected by default. 2.Otherwise, if GGML_CANN_ENABLE_BUF_PRIO_POOL is defined, the priority queue-based memory pool is used. 3.If neither condition is met, the default memory pool is used.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview
Multiple optional memory pools are provided for CANN, including VMM, priority queue-based, and traditional memory pools.
Environment
Benchmark
We use model qwen2.5:0.5b-instruct-fp16.gguf for our benchmark.
./llama.cpp/build/bin/llama-cli -m ds/qwen2.5\:0.5b-instruct-fp16 -p "Building a website can be done in 10 steps:" -ngl 32Use VMM pool
If no environment variables are set, VMM pool is used by default.
Use BUF_PRIO_POOL
If set
export GGML_CANN_DISABLE_VMM_POOL=1 && export GGML_CANN_ENABLE_BUF_PRIO_POOL=1, priority queue-based memory pool will be enabled.BUF_POOL
If set
export GGML_CANN_DISABLE_VMM_POOL=1, traditional buffer pools will be enabled.